Learning method, image identification method, learning device, and image identification system

ABSTRACT

Provided is a learning device that acquires computational imaging information of a computational imaging camera that captures an image with blurring; acquires a normal image captured by a normal camera that captures an image without blurring or an image with blurring smaller than that of the computational imaging camera, and a correct answer label assigned to the normal image; generates an image with blurring based on the computational imaging information and the normal image; and performs machine learning using the image with blurring and the correct answer label to create an image identification model for identifying an image captured by the computational imaging camera.

TECHNICAL FIELD

The present disclosure relates to an image identification method and animage identification system in an environment requiring privacyprotection, such as home or indoor, and a learning method and a learningdevice for creating an image identification model used for the imageidentification.

BACKGROUND ART

Patent Literature 1 below discloses an image identification system inwhich an identifier receives a computational image captured by alight-field camera or the like to identify an object included in thecomputational image using a learned identification model.

The computational image is difficult to be visually recognized by aperson due to blurring that is intentionally created due to an influencesuch as superimposition of multiple images each having a differentviewpoint, or a subject image that is less likely to be focused due tonon-use of a lens. Thus, the computational image is preferably used toconstruct an image identification system in an environment requiringprivacy protection, such as home or indoor.

Unfortunately, a computational image is difficult to be visuallyrecognized by a person, so that it is difficult to assign an accuratecorrect answer label to the computational image captured by alight-field camera or the like, in machine learning for creating anidentification model. As a result, learning efficiency deteriorates.

Patent Literature 1 below takes no measure against this problem, andthus is desired to be improved in learning efficiency by implementingeffective technical measures.

CITATION LIST Patent Literature

-   Patent Literature 1: WO 2019/054092 A

SUMMARY OF INVENTION

It is an object of the present disclosure to provide a technique capableof improving image identification accuracy and improving learningefficiency of machine learning while protecting privacy of a subject inan image identification system.

A learning method according to an aspect of the present disclosureincludes, by an information processing device serving as a learningdevice: acquiring computational imaging information of a first camerathat captures an image with blurring, the computational imaginginformation being a difference image between a first image and a secondimage that are captured by the first camera, the first image including apoint light source in a lighting state and the second image includingthe point light source in a non-lighting state; acquiring a third imagecaptured by a second camera that captures an image without blurring oran image with blurring smaller than that of the first camera, and acorrect answer label assigned to the third image; generating a fourthimage with blurring based on the computational imaging information andthe third image; and performing machine learning using the fourth imageand the correct answer label to create an image identification model foridentifying an image captured by the first camera.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating a configuration of an imageidentification system according to a first embodiment.

FIG. 2 is a flowchart illustrating a procedure of main processing of animage identification system.

FIG. 3 is a diagram schematically illustrating structure of a lenselessmulti-pinhole camera as an example of a computational imaging camera.

FIG. 4A is a diagram illustrating a positional relationship amongmultiple pinholes in a multi-pinhole camera.

FIG. 4B is a diagram illustrating an example of an image captured by amulti-pinhole camera.

FIG. 4C is a diagram illustrating an example of an image captured by amulti-pinhole camera.

FIG. 5 is a flowchart illustrating a procedure of main processing of alearning device.

FIG. 6 is a schematic diagram illustrating a configuration of an imageidentification system according to a second embodiment.

FIG. 7 is a flowchart illustrating a procedure of main processing of animage identification system.

FIG. 8A is a diagram for illustrating creation processing of adifference image.

FIG. 8B is a diagram for illustrating creation processing of adifference image.

FIG. 8C is a diagram for illustrating creation processing of adifference image.

FIG. 9 is a flowchart illustrating a procedure of main processing of acomputational imaging information acquisition unit when a lighttransport matrix (LTM) is used as computational imaging information.

FIG. 10 is a schematic diagram illustrating multiple point spreadfunctions (PSFs).

FIG. 11 is a schematic diagram illustrating a configuration of an imageidentification system according to a third embodiment.

FIG. 12 is a flowchart illustrating a procedure of main processing of animage identification system.

FIG. 13 is a flowchart illustrating a procedure of main processing of animage identification system.

FIG. 14 is a flowchart illustrating a procedure of main processing of animage identification system.

FIG. 15 is a schematic diagram illustrating a configuration of an imageidentification system according to a fourth embodiment.

FIG. 16 is a flowchart illustrating a procedure of main processing of alearning device.

FIG. 17A is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 17B is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 17C is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 17D is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 18A is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 18B is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 18C is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 18D is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 19 is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 20 is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 21 is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 22A is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 22B is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 22C is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 22D is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 22E is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 22F is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 23A is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 23B is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

FIG. 23C is a schematic diagram illustrating a configuration of amulti-pinhole camera according to a modification.

DESCRIPTION OF EMBODIMENTS

(Underlying Knowledge of Present Disclosure)

Various recognition techniques in home, indoor, or the like, such asbehavior recognition of a person in environment and person recognitionof a device operator, are important. In recent years, a technique calleddeep learning has attracted attention for object identification. Thedeep learning is machine learning using a neural network having amultilayer structure, and enables achieving more accurate identificationperformance as compared with a conventional method by using a largeamount of learning data. In such object identification, imageinformation is particularly effective. Various methods have beenproposed for greatly improving conventional object identificationcapability by using a camera as an input device and performing deeplearning using image information as an input.

Unfortunately, disposing a camera in home or the like causes a problemin that privacy is violated when a captured image leaks to the outsidedue to hacking or the like. Thus, a measure is required to protectprivacy of a subject even when a captured image leaks to the outside.

Computational images captured by a light-field camera or the like aredifficult to be visually recognized by a person due to blurring that isintentionally created due to an influence such as superimposition ofmultiple images each having a different viewpoint, or a subject imagethat is less likely to be focused due to non-use of a lens. Thus, thecomputational image is preferably used to construct an imageidentification system in an environment requiring privacy protection,such as home or indoor.

The image identification system disclosed in Patent Literature 1 isconfigured such that a target area is photographed by a light-fieldcamera or the like, and a computational image acquired by thephotographing is input to an identifier. This configuration allows theidentifier to identify an object included in the computational imageusing a learned identification model. When the target area isphotographed by a light-field camera or the like that captures acomputational image as described above, privacy of a subject can beprotected even when the photographed image leaks to the outside due tothe computational image that is difficult to be visually recognized by aperson.

The image identification system disclosed in Patent Literature 1 isconfigured such that the identification model used by the identifier iscreated by performing machine learning using a computational imagecaptured by a light-field camera or the like as learning data.Unfortunately, a computational image is difficult to be visuallyrecognized by a person, so that it is difficult to assign an accuratecorrect answer label to the computational image captured by alight-field camera or the like, in machine learning for creating anidentification model. When an incorrect correct answer label is assignedto the computational image for learning, learning efficiency of themachine learning deteriorates.

To solve such a problem, the present inventors have devised performingmachine learning that uses an image without blurring (hereinafterreferred to as a “normal image”) instead of an image with blurring(hereinafter referred to as a “blurred image”) such as a computationalimage in a stage of accumulating learning data, and that uses a blurredimage obtained by converting a normal image based on computationalimaging information of a used camera in a subsequent learning stage. Asa result, the present inventors have found that image identificationaccuracy and learning efficiency of machine learning can be improvedwhile privacy of a subject is protected, and have conceived the presentdisclosure.

As another viewpoint of privacy protection, it is also important toreduce a psychological load on a user whose image is to be captured byan image recognition device. Capturing a blurred image enables appealfor protecting privacy of a subject. However, when computational imaginginformation is set in a region unrelated to the user, such as a factoryof a manufacturer, a psychological load on the user may increase due toa suspicion that the manufacturer can restore a normal image from ablurred image. Alternatively, the present inventors have considered thatthis psychological load can be reduced when a user himself/herself,whose image is to be captured, can change the computational imaginginformation, and have conceived the present disclosure.

Next, each aspect of the present disclosure will be described.

A learning method according to an aspect of the present disclosureincludes, by an information processing device serving as a learningdevice: acquiring computational imaging information of a first camerathat captures an image with blurring, the computational imaginginformation being a difference image between a first image and a secondimage that are captured by the first camera, the first image including apoint light source in a lighting state and the second image includingthe point light source in a non-lighting state; acquiring a third imagecaptured by a second camera that captures an image without blurring oran image with blurring smaller than that of the first camera, and acorrect answer label assigned to the third image; generating a fourthimage with blurring based on the computational imaging information andthe third image; and performing machine learning using the fourth imageand the correct answer label to create an image identification model foridentifying an image captured by the first camera.

In the present disclosure, the term “blurring” indicates a state inwhich visual recognition by a person is difficult due to an influencesuch as superimposition of multiple images each having a differentviewpoint and being captured by a light-field camera, a lensless camera,or the like, or a subject image that is less likely to be focused due tonon-use of a lens, or a state in which a subject is simply out of focus.The term “image with blurring” means an image that is difficult to bevisually recognized by a person or an image in which a subject is out offocus. The term “large blur” means a large degree of difficulty invisual recognition by a person or a large degree of out-of-focus of asubject, and the term “small blur” means a small degree of thedifficulty in visual recognition or a small degree of the out-of-focus.The term “image without blurring” means an image that is easily visuallyrecognized by a person or an image in which a subject is focused.

This configuration allows a target area where a subject as an imageidentification target is located to be captured by the first camera thatcaptures an image with blurring. Thus, even when an image captured bythe first camera leaks to the outside, the image is difficult to bevisually recognized by a person, so that privacy of the subject can beprotected. The third image serving as learning data is captured by thesecond camera that captures an image without blurring or an image withblurring smaller than that of the first camera. Thus, the image iseasily visually recognized by a person, so that an accurate correctanswer label can be easily assigned to the third image. Thecomputational imaging information of the first camera is a differenceimage between the first image including the point light source in alighting state and the second image including the point light source ina non-lighting state. Thus, the computational imaging information of thefirst camera to be actually used can be accurately acquired withoutbeing affected by the subject other than the point light source. Thisconfiguration enables the fourth image, which is to be used for machinelearning, to be accurately generated based on the computational imaginginformation and the third image. As a result, image identificationaccuracy and learning efficiency of machine learning can be improvedwhile privacy of a subject is protected.

The above aspect may be configured such that the first camera is any oneof a coded aperture camera including a mask having a mask pattern with atransmittance different for each region, a multi-pinhole cameraincluding a mask having a mask pattern in which multiple pinholes areformed and an image sensor with a light receiving surface on which themask is disposed, and a light-field camera that acquires a light fieldfrom a subject.

This configuration enables an image with blurring, which is difficult tobe visually recognized by a person, to be appropriately captured byusing any one of the coded aperture camera, the multi-pinhole camera,and the light-field camera as the first camera.

The above aspect may be configured such that the first camera includesno optical system that forms an image of light from a subject on animage sensor.

This configuration enables blurring to be intentionally created in animage captured by the first camera because the first camera includes nooptical system that forms an image of light from a subject on an imagesensor. As a result, the subject included in the captured image isfurther difficult to be identified, so that an effect of protecting theprivacy of the subject can be further enhanced.

The above aspect may be configured such that the mask is changeable toanother mask having a different mask pattern.

This configuration enables computational imaging information to bechanged for each user by allowing a corresponding user to arbitrarilychange the mask, for example, because the computational imaginginformation of the first camera also changes by changing the mask. As aresult, it is difficult for a third party to inversely convert thefourth image into the third image, so that the effect of protecting theprivacy of the subject can be further enhanced.

The above aspect may be configured such that the computational imaginginformation is any one of a point spread function and a light transportmatrix.

This configuration enables the computational imaging information of thefirst camera to be easily and appropriately acquired by using any one ofthe PSF or the LTM.

The above aspect may be configured such that the information processingdevice performs lighting control of the point light source and imagingcontrol of the first image captured by the first camera, and performsnon-lighting control of the point light source and imaging control ofthe second image captured by the first camera.

This configuration enables timing of turning on or off the point lightsource and timing of imaging with the first camera to be accuratelysynchronized by allowing the information processing device to controloperation of each of the point light source and the first camera.

The above aspect may be configured such that the information processingdevice performs re-imaging control of the first image and the secondimage captured by the first camera when the difference image has imagequality less than an allowable value.

This configuration causes the information processing device to performre-imaging control on the first camera when the difference image has theimage quality less than the allowable value, and thus enables acquiringa difference image using the point light source in which a luminancevalue is appropriately adjusted. As a result, appropriate computationalimaging information of the first camera can be acquired.

The above aspect may be configured such that the information processingdevice corrects at least one of exposure time and gain of the firstcamera in the re-imaging control to cause each of the first image andthe second image to have a maximum luminance value within apredetermined range.

This configuration performs the re-imaging control to correct at leastone of the exposure time and the gain of the first camera, and thusenables acquiring a difference image using the point light source inwhich a luminance value is appropriately adjusted.

An image identification method according to another aspect of thepresent disclosure includes, in an identification device including anidentification unit: allowing the identification unit to receive animage captured by a first camera that captures an image with blurring;identifying, by the identification unit, the received image based on alearned image identification model; and outputting a result of theidentification of the identification unit, wherein the imageidentification model is the image identification model created by thelearning method according to the above aspect.

This configuration allows a target area where a subject as an imageidentification target is located to be captured by the first camera thatcaptures an image with blurring. Thus, even when an image captured bythe first camera leaks to the outside, the image is difficult to bevisually recognized by a person, so that privacy of the subject can beprotected. The third image serving as learning data is captured by thesecond camera that captures an image without blurring or an image withblurring smaller than that of the first camera. Thus, the image iseasily visually recognized by a person, so that an accurate correctanswer label can be easily assigned to the third image. Thecomputational imaging information of the first camera is a differenceimage between the first image including the point light source in alighting state and the second image including the point light source ina non-lighting state. Thus, the computational imaging information of thefirst camera to be actually used can be accurately acquired withoutbeing affected by the subject other than the point light source. Thisconfiguration enables the fourth image, which is to be used for machinelearning, to be accurately generated based on the computational imaginginformation and the third image. As a result, image identificationaccuracy and learning efficiency of machine learning can be improvedwhile privacy of a subject is protected.

A learning device according to yet another aspect of the presentdisclosure includes: an acquisition unit that acquires computationalimaging information of a first camera that captures an image withblurring, the computational imaging information being a difference imagebetween a first image and a second image that are captured by the firstcamera, the first image including a point light source in a lightingstate and the second image including the point light source in anon-lighting state; a storage unit that stores a third image captured bya second camera that captures an image without blurring or an image withblurring smaller than that of the first camera, and a correct answerlabel assigned to the third image; an image generator that generates afourth image with blurring based on the computational imaginginformation acquired by the acquisition unit and the third image readout from the storage unit; and a learning unit that performs machinelearning using the fourth image generated by the image generator and thecorrect answer label read out from the storage unit to create an imageidentification model for identifying an image captured by the firstcamera.

This configuration allows a target area where a subject as an imageidentification target is located to be captured by the first camera thatcaptures an image with blurring. Thus, even when an image captured bythe first camera leaks to the outside, the image is difficult to bevisually recognized by a person, so that privacy of the subject can beprotected. The third image serving as learning data is captured by thesecond camera that captures an image without blurring or an image withblurring smaller than that of the first camera. Thus, the image iseasily visually recognized by a person, so that an accurate correctanswer label can be easily assigned to the third image. Thecomputational imaging information of the first camera is a differenceimage between the first image including the point light source in alighting state and the second image including the point light source ina non-lighting state. Thus, the computational imaging information of thefirst camera to be actually used can be accurately acquired withoutbeing affected by the subject other than the point light source. Thisconfiguration enables an image synthesizer to accurately generate thefourth image, which is to be used for machine learning, based on thecomputational imaging information and the third image. As a result,image identification accuracy and learning efficiency of machinelearning can be improved while privacy of a subject is protected.

An image identification system according to yet another aspect of thepresent disclosure includes: an acquisition unit that acquirescomputational imaging information of a first camera that captures animage with blurring, the computational imaging information being adifference image between a first image and a second image that arecaptured by the first camera, the first image including a point lightsource in a lighting state and the second image including the pointlight source in a non-lighting state; a storage unit that stores a thirdimage captured by a second camera that captures an image withoutblurring or an image with blurring smaller than that of the firstcamera, and a correct answer label assigned to the third image; an imagegenerator that generates a fourth image with blurring based on thecomputational imaging information acquired by the acquisition unit andthe third image read out from the storage unit; a learning unit thatperforms machine learning using the fourth image generated by the imagegenerator and the correct answer label read out from the storage unit tocreate an image identification model; an identification unit thatidentifies an image captured by the first camera based on the imageidentification model created by the learning unit; and an output unitthat outputs an identification result of the identification unit.

This configuration allows a target area where a subject as an imageidentification target is located to be captured by the first camera thatcaptures an image with blurring. Thus, even when an image captured bythe first camera leaks to the outside, the image is difficult to bevisually recognized by a person, so that privacy of the subject can beprotected. The third image serving as learning data is captured by thesecond camera that captures an image without blurring or an image withblurring smaller than that of the first camera. Thus, the image iseasily visually recognized by a person, so that an accurate correctanswer label can be easily assigned to the third image. Thecomputational imaging information of the first camera is a differenceimage between the first image including the point light source in alighting state and the second image including the point light source ina non-lighting state. Thus, the computational imaging information of thefirst camera to be actually used can be accurately acquired withoutbeing affected by the subject other than the point light source. Thisconfiguration enables an image synthesizer to accurately generate thefourth image, which is to be used for machine learning, based on thecomputational imaging information and the third image. As a result,image identification accuracy and learning efficiency of machinelearning can be improved while privacy of a subject is protected.

The present disclosure can also be implemented as a computer program forcausing a computer to execute each characteristic configuration includedin a method as described above, or can also be implemented as a deviceor a system that operates based on the computer program. It is needlessto say that such a computer program can be distributed as acomputer-readable non-volatile recording medium such as a CD-ROM, or canbe distributed via a communication network such as the Internet.

Each of the embodiments described below illustrates a specific exampleof the present disclosure. Numerical values, shapes, components, steps,order of steps, and the like shown in the following embodiments aremerely examples, and are not intended to limit the present disclosure.The components in the embodiments below include a component that is notdescribed in an independent claim representing the highest concept andthat is described as an arbitrary component. All the embodiments haverespective contents that can be combined.

Hereinafter, the embodiments of the present disclosure will be describedin detail with reference to the drawings. Elements denoted by the samecorresponding reference numerals in different drawings represent thesame or corresponding elements.

First Embodiment

FIG. 1 is a schematic diagram illustrating a configuration of an imageidentification system 10 according to a first embodiment of the presentdisclosure. The image identification system 10 includes a learningdevice 20 and an identification device 30. The identification device 30includes a computational imaging camera 101, an identification unit 106,and an output unit 107. The identification unit 106 includes a processorsuch as a CPU, and a memory such as a semiconductor memory. The outputunit 107 is a display device, a speaker, or the like. The learningdevice 20 includes a learning database 102, a computational imaginginformation acquisition unit 103, a database correction unit 104, and alearning unit 105. The learning database 102 is a storage unit such asan HDD, an SSD, or a semiconductor memory. The computational imaginginformation acquisition unit 103, the database correction unit 104, andthe learning unit 105 are each a processor such as a CPU.

FIG. 2 is a flowchart illustrating a procedure of main processing of theimage identification system 10. The flowchart illustrates a flow ofimage identification processing of the identification device 30. Thecomputational imaging camera 101 first photographs a target area, andinputs a computational image obtained by the photographing into theidentification unit 106 (step S101). The identification unit 106subsequently identifies the computational image using the learned imageidentification model (step S102). This image identification model is theimage identification model created by learning of the learning device20. Next, the output unit 107 outputs a result of identification of theidentification unit 106. Details of the processing in each step will bedescribed later.

Unlike a normal camera that captures a normal image without blurring,the computational imaging camera 101 captures a computational imageserving as an image with blurring. Although a subject in thecomputational image cannot be recognized by a person who views thecomputational image itself due to intentionally created blurring, animage can be generated from the computational image by performing imageprocessing on the captured computational image, the image being able tobe recognized by the person or identified by the identification unit106.

FIG. 3 is a diagram schematically illustrating structure of a lenselessmulti-pinhole camera 301 as an example of the computational imagingcamera 101. The multi-pinhole camera 301 illustrated in FIG. 3 includesa multi-pinhole mask 301 a, and an image sensor 301 b such as a CMOS.The multi-pinhole mask 301 a is disposed at a predetermined intervalfrom a light receiving surface of the image sensor 301 b. Themulti-pinhole mask 301 a has multiple pinholes 301 aa disposed at randomor equal intervals. The multiple pinholes 301 aa are also referred to asmulti-pinholes. The image sensor 301 b acquires an image of a subject302 through each pinhole 301 aa. The image acquired through a pinhole isreferred to as a pinhole image.

The pinhole image of the subject 302 differs depending on a position anda size of each pinhole 301 aa, so that the image sensor 301 b acquires asuperimposed image (multiple image) in a state in which multiple pinholeimages are superimposed while being slightly shifted. The multiplepinholes 301 aa has a positional relationship that affects

a positional relationship among the multiple pinhole images projected onimage sensor 301 b (i.e., a degree of superimposition of multipleimages), and a size of each pinhole 301 aa affects a degree of blurringof a pinhole image.

Using the multi-pinhole mask 301 a enables acquiring multiple pinholeimages each having a different position and a different degree ofblurring while superimposing the images. That is, a computational imagein which multiple images and blurring are intentionally created can beacquired. Thus, a photographed image is a multiple image and a blurredimage, and an image in which privacy of the subject 302 is protected bythe blurring can be acquired. When the pinholes are changed in number,position, and size, images each having a different blurring pattern canbe acquired. That is, the multi-pinhole mask 301 a may be configured tobe able to be easily attached and detached by a user, and multiple typesof multi-pinhole mask 301 a, each having a different mask pattern, maybe prepared in advance to allow the user to freely replace themulti-pinhole mask 301 a to be used.

Changing a mask as described above can be implemented by various methodsin addition to the replacement of the mask, such as:

-   -   a user arbitrarily rotating the mask rotatably attached in front        of an image sensor;    -   the user making a hole in an arbitrary place of the plate        attached in front of the image sensor;    -   using a liquid crystal mask or the like using a spatial light        modulator or the like to arbitrarily set transmittance at each        position in the mask; and    -   molding a mask using a stretchable material such as rubber to        change a position and a size of a hole by physically deforming        the mask by application of an external force. Hereinafter, these        modifications will be described in order.

<Modification in Which User Arbitrarily Rotates Mask>

FIGS. 17A to 17D are each a schematic diagram illustrating aconfiguration of the multi-pinhole camera 301 in which a user canarbitrarily rotate a mask. FIG. 17A illustrates an overview of themulti-pinhole camera 301 in which a user can arbitrarily rotate a mask,and FIG. 17B is a schematic diagram illustrating a section of themulti-pinhole camera 301. The multi-pinhole camera 301 includes themulti-pinhole mask 301 a rotatable with respect to a housing 401, and agripper 402 connected to the multi-pinhole mask 301 a. The user can fixor rotate the multi-pinhole mask 301 a with respect to the housing 401by gripping and operating the gripper 402. This kind of mechanism may beconfigured as follows: a screw is provided in the gripper 402; the screwis tightened to fix the multi-pinhole mask 301 a; and the screw isloosened to allow the multi-pinhole mask 301 a to be rotatable. FIGS.17C and 17D are schematic diagrams illustrating the multi-pinhole mask301 a that rotates by 90 degrees when the gripper 402 is rotated by 90degrees. As described above, the multi-pinhole mask 301 a can be rotatedwhen the user moves the gripper 402.

The multi-pinhole camera 301, in which the user can arbitrarily rotatethe mask, may be configured such that the multi-pinhole mask 301 aincludes pinholes disposed asymmetrically with respect to a rotationaxis as illustrated in FIG. 17C. This configuration enables variousmulti-pinhole patterns to be implemented when the user rotates the mask.

As a matter of course, the multi-pinhole camera 301, in which the usercan arbitrarily rotate the mask, may be configured without the gripper402. FIGS. 18A and 18B are each a schematic diagram illustrating anotherconfiguration example of the multi-pinhole camera 301 in which a usercan arbitrarily rotate a mask. FIG. 18A illustrates an overview of theother example of the multi-pinhole camera 301 in which a user canarbitrarily rotate a mask, and FIG. 18B is a schematic diagramillustrating a section of the other example. The multi-pinhole mask 301a is fixed to a lens barrel 411. Then, the image sensor 301 b isinstalled in another lens barrel 412, and the lens barrel 411 and thelens barrel 412 are rotatable with a screw configuration. That is, thelens barrel 412 is provided outside the lens barrel 411, and a malethread is cut outside the lens barrel 411 serving as a joint part, and afemale thread is cut inside the lens barrel 412. The lens barrel 411includes the male thread to which a fixture 413 is first attached, andthen the lens barrel 412 is attached. As with the lens barrel 412, afemale thread is cut also in the fixture 413. This configurationdescribed above enables a rotation angle of the multi-pinhole camera 301to be changed due to change in screwing depth caused by a screwingposition of the fixture 413 into the lens barrel 411 when the lensbarrel 411 is screwed into the lens barrel 412.

FIGS. 18C and 18D are schematic diagrams illustrating that the screwingdepth changes and the rotation angle of the multi-pinhole camera 301changes in accordance with a screwing position of the fixture 413 intothe lens barrel 411. FIG. 18C is a schematic diagram when the fixture413 is screwed onto the lens barrel 411 all the way, and FIG. 18D is aschematic diagram when the fixture 413 is screwed onto the lens barrel411 only midway. As illustrated in FIG. 18C, when the fixture 413 isscrewed onto the lens barrel 411 all the way, the lens barrel 412 can bescrewed onto the lens barrel 411 all the way. In contrast, when thefixture 413 is screwed onto the lens barrel 411 only midway, the lensbarrel 412 can be screwed onto the lens barrel 411 only midway, asillustrated in FIG. 18D. Thus, the screwing depth changes in accordancewith the screwing position of the fixture 413 onto the lens barrel 411,and then the rotation angle of the multi-pinhole mask 301 a can bechanged.

<Modification in Which User Makes Hole in Mask>

FIG. 19 is a schematic diagram of a section of the multi-pinhole camera301 in which a user can make a hole in an arbitrary place of a mask 301ab attached in front of the image sensor 301 b. FIG. 19 illustrates thesame components as those in FIG. 17 that are denoted by the samecorresponding reference numerals as those in FIG. 17 , and that are notdescribed. The mask 301 ab initially has no pinhole. When a user makesmultiple holes in the mask 301 ab at arbitrary positions using a needleor the like, a multi-pinhole mask in an arbitrary shape can be created.

<Modification in which Transmittance of Each Position in Mask isArbitrarily Set Using Spatial Light Modulator>

FIG. 20 is a schematic diagram of a section of the multi-pinhole camera301 configured to arbitrarily set transmittance at each position in amask using a spatial light modulator 420. FIG. 20 illustrates the samecomponents as those in FIG. 19 that are denoted by the samecorresponding reference numerals as those in FIG. 19 , and that are notdescribed. The spatial light modulator 420 is composed of a liquidcrystal or the like, and can change the transmittance for each pixel.The spatial light modulator 420 functions as a multi-pinhole mask.Change of the transmittance can be controlled by a spatial lightmodulator controller (not illustrated). Thus, when the user selects anarbitrary pattern from multiple transmittance patterns prepared inadvance, various mask patterns (multi-pinhole patterns) can beimplemented.

<Modification in which Mask is Deformed by Application of ExternalForce>

FIGS. 21 and 22A to 22F are each a schematic diagram of a section of themulti-pinhole camera 301 configured to deform a mask by application ofan external force. FIG. 21 illustrates the same components as those inFIG. 19 that are denoted by the same corresponding reference numerals asthose in FIG. 19 , and that are not described. A multi-pinhole mask 301ac includes multiple masks 301 a 1, 301 a 2, and 301 a 3, and each maskhas a drive unit (not illustrated) that independently applies anexternal force. FIGS. 22A to 22C are respectively schematic diagrams forillustrating the three masks 301 a 1, 301 a 2, and 301 a 3 constitutingthe multi-pinhole mask 301 ac. Here, each mask has a shape in which afan shape and a circular ring are combined. As a matter of course, thisconfiguration is an example, the shape is not limited to the fan shape,and the number of components is not limited to three. Each mask isprovided with one or more pinholes. The mask may be provided with nopinhole. The mask 301 a 1 is provided with two pinholes 301 aa 1 and 301aa 2, the mask 301 a 2 is provided with one pinhole 301 aa 3, and themask 301 a 3 is provided with two pinholes 301 aa 4 and 301 aa 5. Whenthese three masks 301 a 1 to 301 a 3 are moved by application of anexternal force, various multi-pinhole patterns can be created.

FIGS. 22D to 22F illustrate respective three types of multi-pinhole mask301 ac composed of the three masks 301 a 1 to 301 a 3. When each of themasks 301 a 1 to 301 a 3 is moved in a different mode by correspondingone of drive units (not illustrated), a mask having five pinholes isfaulted in each of FIGS. 22D and 22E, and a mask having four pinholes isformed in FIG. 22F. This kind of drive unit for a mask can be fabricatedby using an ultrasonic motor or a linear motor widely used in autofocusor the like. As described above, the multi-pinhole mask 301 ac can bechanged in number and position of pinholes by application of an externalforce.

As a matter of course, the multi-pinhole mask may be changed not only innumber and position of pinholes but also in size thereof. FIGS. 23A to23C are each a schematic diagram for illustrating a configuration of amulti-pinhole mask 301 ad in the multi-pinhole camera 301 configured todeform a mask using application of an external force. The multi-pinholemask 301 ad is made of an elastic material and includes multiplepinholes, and four drive units 421 to 424 capable of independentlycontrolling respective four corners. As a matter of course, the numberof drive units does not need to be four. When each of the drive units421 to 424 is driven, the pinholes in the multi-pinhole mask 301 ad canbe each changed in position and size.

FIG. 23B is a schematic diagram illustrating a state where the driveunits 421 to 424 are driven in the same direction. FIG. 23B illustratesarrows in the respective drive units 421 to 424, the arrows indicatingdriven directions of the respective drive units. In this case, themulti-pinhole mask 301 ad is displaced parallel in the driving directionof the drive units. Then, FIG. 23C is a schematic diagram illustrating astate in which the drive units 421 to 424 are each driven outward from acentral part of the multi-pinhole mask 301 ad. In this case, themulti-pinhole mask 301 ad is stretched in accordance with elasticity, sothat the pinholes are each increased in size. The drive units 421 to 424described above can be fabricated by using an ultrasonic motor or alinear motor widely used in autofocus or the like. As described above,the multi-pinhole mask 301 ac can be changed in position and size ofpinholes by application of an external force.

FIG. 4A is a diagram illustrating a positional relationship amongmultiple pinholes 301 aa in the multi-pinhole camera 301. This exampleshows three pinholes 301 aa provided linearly. An interval between aleftmost pinhole 301 aa and a center pinhole 301 aa is set to L1, and aninterval between the center pinhole 301 aa and a rightmost pinhole 301aa is set to L2 (<L1).

FIGS. 4B and 4C are each a diagram illustrating an example of an imagecaptured by the multi-pinhole camera 301. FIG. 4B illustrates an exampleof a captured image with a small subject image due to a relatively longdistance between the multi-pinhole camera 301 and the subject 302. FIG.4C illustrates an example of a captured image with a large subject imagedue to a relatively short distance between the multi-pinhole camera 301and the subject 302. When the intervals L1 and L2 are made differentfrom each other, multiple images having different viewpoints aresuperimposed regardless of the distance between the multi-pinhole camera301 and the subject 302, thereby capturing a superimposed image withmultiple subject images that are superimposed in an individuallyunrecognizable manner.

Examples of a well-known camera other than the multi-pinhole camera 301,which are available for the computational imaging camera 101, include:

a coded aperture camera with a mask having a mask pattern with atransmittance different for each region, the mask being disposed betweenan image sensor and a subject;

a light-field camera having a configuration in which a microlens arrayis disposed on a light receiving surface of an image sensor to acquire alight field; and

a compression sensing camera that captures an image by weighting andadding pixel information of time and space.

The computational imaging camera 101 desirably does not include anoptical system, such as a lens, a prism, and a mirror, for forming animage of light from a subject on an image sensor. Eliminating theoptical system enables reducing the camera in size, weight, and cost,and improving a design, and also intentionally creating blurring in animage captured by the camera.

The identification unit 106 uses the image identification model that isa learning result of the learning device 20 to identify categoryinformation on subjects such as a person (including a behavior and anexpression), an automobile, a bicycle, and a traffic light, andpositional information on each of the subjects, which are included in animage of a target area captured by the computational imaging camera 101.Machine learning such as deep learning using a multilayer neural networkmay be used for learning for creating the image identification model.

The output unit 107 outputs an identification result of theidentification unit 106. The output unit 107 may include an interfaceunit to present the identification result to the user by an image, text,voice, or the like, or may include an apparatus controller to change acontrol method depending on the identification result.

The learning device 20 includes the learning database 102, thecomputational imaging information acquisition unit 103, the databasecorrection unit 104, and the learning unit 105. The learning device 20performs learning for creating the image identification model to be usedby the identification unit 106 in association with the computationalimaging information of the computational imaging camera 101 that is tobe actually used for capturing an image of the target area.

FIG. 5 is a flowchart illustrating a procedure of main processing of thelearning device 20 of the image identification system 10.

The computational imaging information acquisition unit 103 firstacquires computational imaging information that indicates a mode ofburring expressing what type of blurred image is captured by thecomputational imaging camera 101 (step S201). In step S201, thecomputational imaging camera 101 may include a transmitter, and thecomputational imaging information acquisition unit 103 may include areceiver, thereby exchanging the computational imaging information of awired or wireless manner. Alternatively, the computational imaginginformation acquisition unit 103 may include an interface, and the usermay input the computational imaging information to the computationalimaging information acquisition unit 103 via the interface.

For example, when the computational imaging camera 101 is themulti-pinhole camera 301, a point spread function (PSF) indicating astate of two-dimensional computational imaging may be used as thecomputational imaging information. The PSF is a transfer function of acamera such as a multi-pinhole camera or a coded aperture camera, and isexpressed by the following relationship.

y=k*x

Here, y is a computational image with blurring captured by themulti-pinhole camera 301, k is a PSF, and x is a normal image withoutblurring of a scene captured by a normal camera. Then, * is aconvolution operator.

Alternatively, a light transport matrix (LTM) indicating computationalimaging information including four or more dimensions (two or moredimensions on a camera side and two or more dimensions on a subjectside) may be used as the computational imaging information instead ofthe PSF. The LTM is a transfer function used in a light-field camera.

For example, when the computational imaging camera 101 is themulti-pinhole camera 301, the PSF can be acquired by photographing apoint light source with the multi-pinhole camera 301. This can be seenfrom the fact that the PSF corresponds to an impulse response of thecamera. That is, a captured image itself of the point light sourceobtained by capturing an image of the point light source with themulti-pinhole camera 301 is the PSF as the computational imaginginformation of the multi-pinhole camera 301. Here, a difference imagebetween a lighting state and a non-lighting state is desirably used asthe captured image of the point light source, and this will be describedin a second embodiment described later.

Next, the database correction unit 104 acquires a normal image withoutblurring included in the learning database 102, and the learning unit105 acquires annotation information included in the learning database102 (step S202).

Subsequently, the database correction unit 104 (image generator)corrects the learning database 102 using the computational imaginginformation acquired by the computational imaging informationacquisition unit 103 (step S203). For example, when the identificationunit 106 identifies a behavior of a person in an environment, thelearning database 102 holds multiple normal images without blurringphotographed by a normal camera, and annotation information (correctanswer label) that is assigned to each image and that indicates aposition at which the person has performed what kind of behavior in theimage. When a normal camera is used, annotation information may beassigned to an image captured by the camera. However, when acomputational image is acquired by a multi-pinhole camera or alight-field camera, it is difficult to assign annotation information tothe image because a person cannot find what the image shows even whenlooking at the image. Additionally, even when learning processing isperforated on an image captured by a normal camera significantlydifferent from the computational imaging camera 101, the identificationunit 106 does not increase in identification accuracy. Thus, theidentification accuracy is improved as follows: a database in whichannotation information is preliminarily assigned to an image captured bya normal camera is held as the learning database 102; only the capturedimage is deformed in accordance with the computational imaginginformation of the computational imaging camera 101 to create learningdata set corresponding to the computational imaging camera 101; and thelearning processing is performed. For this processing, the databasecorrection unit 104 calculates a corrected image y below using the PSF,which is the computational imaging information acquired by thecomputational imaging information acquisition unit 103, for an image zthat is photographed by the normal camera and that is prepared inadvance.

y=k*z

Here, k represents the PSF that is the computational imaging informationacquired by the computational imaging information acquisition unit 103,and * represents a convolution operator.

The learning unit 105 performs the learning processing using thecorrected image calculated by the database correction unit 104 and theannotation information acquired from the learning database 102 (stepS204). For example, when the identification unit 106 is constructed by amultilayer neural network, machine learning by deep learning isperformed using the corrected image and the annotation information asteacher data. As a prediction error correction algorithm, a backpropagation method or the like may be used. As a result, the learningunit 105 creates an image identification model for the identificationunit 106 to identify an image captured by the computational imagingcamera 101. The corrected image matches the computational imaginginformation of the computational imaging camera 101, so that thelearning described above enables learning suitable for the computationalimaging camera 101 to allow the identification unit 106 to performidentification processing with high accuracy.

The image identification system 10 according to the present embodimentallows an image of a target area, where the subject 302 as an imageidentification target is located, to be captured by the computationalimaging camera 101 (first camera) that captures a computational imagethat is a blurred image. Thus, even when an image captured by thecomputational imaging camera 101 leaks to the outside, the computationalimage is difficult to be visually recognized by a person, so thatprivacy of the subject 302 can be protected. Then, a normal image (thirdimage) to be accumulated in the learning database 102 is captured by anormal camera (second camera) that captures an image without blurring(or an image with blurring smaller than that of a computational image).Thus, the image is easily visually recognized by a person, so thataccurate annotation information (correct answer label) can be easilyassigned to the normal image. As a result, image identification accuracyand learning efficiency of machine learning can be improved whileprivacy of the subject 302 is protected.

Using any one of the coded aperture camera, the multi-pinhole camera,and the light-field camera as the computational imaging camera 101enables an image with blurring, which is difficult to be visuallyrecognized by a person, to be appropriately captured.

Eliminating an optical system that forms an image of light from thesubject 302 on the image sensor 301 b in the computational imagingcamera 101 enables blurring to be intentionally created in an imagecaptured by the computational imaging camera 101. As a result, thesubject 302 included in the captured image is further difficult to beidentified, so that an effect of protecting the privacy of the subject302 can be further enhanced.

When the multi-pinhole mask 301 a to be used is configured to be freelychangeable by a user, changing the mask causes the computational imaginginformation of the computational imaging camera 101 to be also changed.Thus, when each of users arbitrarily changes the mask, for example, thecomputational imaging information can be made different for each of theusers. As a result, it is difficult for a third party to inverselyconvert the corrected image (fourth image) into the formal image (thirdimage), so that the effect of protecting the privacy of the subject 302can be further enhanced.

Using any one of the PSF and the LTM as the computational imaginginformation enables the computational imaging information of thecomputational imaging camera 101 to be easily and appropriatelyacquired.

Second Embodiment

FIG. 6 is a schematic diagram illustrating a configuration of an imageidentification system 11 according to a second embodiment of the presentdisclosure. FIG. 6 illustrates the same components as those in FIG. 1that are denoted by the same corresponding reference numerals as thosein FIG. 1 , and that are not described. The image identification system11 includes a learning device 21 including a controller 108. The imageidentification system 11 also includes a light emitter 109 existing in atarget area (environment) that is to be photographed by a computationalimaging camera 101. The light emitter 109 is a light source that can beregarded as a point light source existing in the environment, and is anLED mounted on an electric apparatus or an LED for illumination, forexample. Additionally, only a part of light of a monitor such as an LEDmonitor may be turned on and off to function as the light emitter 109.The controller 108 controls the light emitter 109 and the computationalimaging camera 101 to allow a computational imaging informationacquisition unit 103 to acquire computational imaging information.

FIG. 7 is a flowchart illustrating a procedure of main processing of theimage identification system 11. The flowchart illustrates a flow ofprocessing in which the computational imaging information acquisitionunit 103 acquires computational imaging information of the computationalimaging camera 101.

The controller 108 first issues a lighting instruction to the lightemitter 109 existing in the environment (step S111).

Next, the light emitter 109 turns on lighting according to theinstruction from the controller 108 (step S112).

Subsequently, the controller 108 instructs the computational imagingcamera 101 to capture an image (step S113). This processing enables thelight emitter 109 and the computational imaging camera 101 to operate insynchronism with each other.

Next, the computational imaging camera 101 captures an image accordingto the instruction of the controller 108 (step S114). The captured image(first image) is input from the computational imaging camera 101 intothe computational imaging information acquisition unit 103, and istemporarily held by the computational imaging information acquisitionunit 103.

Subsequently, the controller 108 issues a turn-off instruction to thelight emitter 109 (step S115).

Next, the light emitter 109 turns off the lighting according to theinstruction from the controller 108 (step S116).

Subsequently, the controller 108 instructs the computational imagingcamera 101 to capture an image (step S117).

Next, the computational imaging camera 101 captures an image accordingto the instruction of the controller 108 (step S118). The captured image(second image) is input from the computational imaging camera 101 intothe computational imaging information acquisition unit 103.

Subsequently, the computational imaging information acquisition unit 103creates a difference image between the first image and the second image(step S119). Acquiring the difference image between the first image whenthe light emitter 109 is turned on and the second image when the lightemitter 109 is turned off as described above enables acquiring a PSFbeing an image of only the light emitter 109 in a lighting state withoutbeing affected by another subject in the environment.

Next, the computational imaging information acquisition unit 103acquires the created difference image as computational imaginginformation of the computational imaging camera 101 (step S120).

When the PSF is used as the computational imaging information asdescribed above, the computational imaging camera 101 captures twoimages of a scene in which the light emitter 109 is turned on and ascene in which the light emitter 109 is turned off. The image in alighting state and the image in a non-lighting state at this time aredesirably captured in a time difference as little as possible.

FIGS. 8A to 8C are each a diagram for illustrating creation processingof a difference image. FIG. 8A is an image captured by the computationalimaging camera 101 when the light emitter 109 is turned on. It can beseen that the light emitter 109 has a high luminance value. FIG. 8B isan image captured by the computational imaging camera 101 when the lightemitter 109 is turned off. It can be seen that the light emitter 109 hasa lower luminance value than that when the light emitter 109 is turnedon. FIG. 8C illustrates a difference image obtained by subtracting FIG.8B, which is the image captured by the computational imaging camera 101when the light emitter 109 is turned off, from FIG. 8A, which is theimage captured by the computational imaging camera 101 when the lightemitter 109 is turned off. Because only the light emitter 109 serving asa point light source is photographed without being affected by a subjectother than the light emitter 109, it can be seen that the PSF can beacquired.

When an LTM is used as computational imaging information, multiple lightemitters 109 dispersedly disposed in the environment may be used toacquire PSFs at multiple positions, and the PSFs may be used as the LTM.

FIG. 9 is a flowchart illustrating a procedure of main processing of thecomputational imaging information acquisition unit 103 when an LTM isused as computational imaging information. First, PSFs corresponding tothe respective light emitters 109 are acquired (step S301). The PSFseach may be acquired by using a difference image between an image whencorresponding one of the light emitters 109 is turned on and an imagewhen it is turned off as described above. This processing enablesacquiring the PSFs at respective multiple positions on the image. FIG.10 is a schematic diagram illustrating multiple PSFs acquired asdescribed above. This example shows that the PSFs are acquired at sixpoints on the image.

The computational imaging information acquisition unit 103 calculatesthe PSFs in all pixels of the image by performing interpolationprocessing on the multiple PSFs acquired as described above, and setsthe PSFs as the LTM (step S302). The interpolation processing above mayuse general image processing such as morphing. The light emitter 109 maybe a light of a smartphone or a mobile phone of a user. In this case,the user may turn on or off the light emitter 109 instead of thecontroller 108.

When the LTM is used as the computational imaging information, a smallnumber of light emitters 109 may be used to change the light emitters109 in position by moving them instead of disposing a large number oflight emitters 109. This processing may be implemented by using a lightof a smartphone or a mobile phone as the light emitter 109, and turningon and off the light while a user changes its location, for example.Alternatively, an LED mounted on a moving body such as a drone or avacuum cleaner robot may be used. Additionally, the light emitter 109 onthe computational image may be changed in position by installing thecomputational imaging camera 101 on a moving body or the like, orallowing a user to change a direction and a position of thecomputational imaging camera 101.

The image identification system 11 according to the present embodimentis configured such that the computational imaging information of thecomputational imaging camera 101 (first camera) is a difference imagebetween the first image including the point light source in a lightingstate and the second image including the point light source in anon-lighting state. Thus, the computational imaging information of thecomputational imaging camera 101 to be actually used can be accuratelyacquired without being affected by the subject other than the pointlight source. This configuration enables the corrected image (fourthimage), which is to be used for machine learning, to be accuratelygenerated based on the computational imaging information and the normalimage (third image).

Additionally, the controller 108 of the learning device 21 controlsoperation of the light emitter 109 and the computational imaging camera101 to enable timing of turning on or off the light emitter 109 andtiming of imaging with the computational imaging camera 101 to beaccurately synchronized.

Third Embodiment

FIG. 11 is a schematic diagram illustrating a configuration of an imageidentification system 12 according to a third embodiment of the presentdisclosure. FIG. 11 illustrates the same components as those in FIG. 6that are denoted by the same corresponding reference numerals as thosein FIG. 6 , and that are not described. The image identification system12 includes a learning device 22 including a computational imaginginformation determination unit 110. The computational imaginginformation determination unit 110 determines a state of image qualityof computational imaging information acquired by a computational imaginginformation acquisition unit 103. The learning device 22 switchescontents of processing depending on a determination result of thecomputational imaging information determination unit 110.

FIG. 12 is a flowchart illustrating a procedure of main processing ofthe image identification system 12. The flowchart illustrates a flow ofprocessing before and after processing of determining image quality withthe computational imaging information determination unit 110.

First, the computational imaging information acquisition unit 103creates a difference image between a first image when a light emitter109 is turned on and a second image when the light emitter is turned offby a method similar to that in step S119 (FIG. 7 ) of the secondembodiment (step S121).

Next, the computational imaging information determination unit 110determines whether image quality of the difference image created by thecomputational imaging information acquisition unit 103 is an allowablevalue or more (step S122). The difference image between an image in alighting state and an image in a non-lighting state is used becausenothing other than a point light source needs to be shown in a PSF.Unfortunately, when there is a change in scene such as a large movementof a person or a dramatic change in brightness in environment betweenphotographing in a lighting state and photographing in a non-lightingstate, the change appears in the difference image, and thus an accuratePSF cannot be acquired. Thus, the computational imaging informationdetermination unit 110 counts the number of pixels having luminance of acertain value or more in the difference image to determine that the PSFhas image quality less than the allowable value when the number ofpixels is equal to or more than a threshold, and determine that the PSFhas image quality equal to or more than the allowable value when thenumber of pixels is less than the threshold.

When the computational imaging information determination unit 110determines that the difference image has image quality less than theallowable value (step S122: NO), next, the controller 108 instructs thelight emitter 109 to emit light and turn off the light, and instructsthe computational imaging camera 101 to capture an image again (stepS123). In contrast, when the computational imaging informationdetermination unit 110 determines that the difference image has imagequality equal to or higher than the allowable value (step S122: YES),next, the database correction unit 104 corrects a learning database 102using the computational imaging information (PSF) acquired as thedifference image by the computational imaging information acquisitionunit 103 (step S124).

Here, it is conceivable that inappropriate setting of the computationalimaging camera 101 is one of causes of deterioration in image quality ofthe difference image. For example, too short exposure time of thecomputational imaging camera 101 or too small gain of signalamplification causes not only an image to be dark as a whole, but alsoluminance of the light emitter 109 to be buried in noise. Conversely,too long exposure time of the computational imaging camera 101 or toolarge gain of the signal amplification causes not only a high luminanceregion in an image to have a luminance value exceeding an upper limitvalue of a sensing range but also the luminance to be saturated, andthus causing a so-called overexposure state around the light emitter109. Thus, the computational imaging information determination unit 110may check a maximum luminance value of each image when the light emitter109 is turned on and off to determine that the difference image hasimage quality less than the allowable value when the maximum luminancevalue exceeds the upper limit value or is less than a lower limit value,or when the maximum luminance value is out of a predetermined range. Thecomputational imaging information determination unit 110 determines theimage quality of the difference image based on whether the image whenthe light emitter 109 is turned on has a maximum luminance valueexceeding the upper limit value, and thus enabling determination whetherthe light emitter 109 has luminance exceeding the sensing range and theluminance is saturated. The computational imaging informationdetermination unit 110 also determines the image quality of thedifference image based on whether the image when the light emitter 109is turned on has a maximum luminance value less than the lower limitvalue, and thus enabling determination whether the light emitter 109 hasluminance buried in noise. When it is determined that the light emitter109 has luminance saturated or buried in noise, the controller 108 maycontrol the computational imaging camera 101 to change its setting sothat a maximum luminance value falls within the predetermined range tocapture an image again.

FIG. 13 is a flowchart illustrating a procedure of main processing ofthe image identification system 12. The flowchart illustrates a flow ofprocessing before and after processing of determining image quality withthe computational imaging information determination unit 110.

First, the computational imaging information acquisition unit 103acquires the first image captured by the computational imaging camera101 when the light emitter 109 is turned on (step S131).

Next, the computational imaging information determination unit 110checks whether the first image acquired by the computational imaginginformation acquisition unit 103 has a maximum luminance value exceedingan upper limit value Th1, thereby determining whether the luminance ofthe image is saturated (step S132).

When the maximum luminance value exceeds the upper limit value Th1, orwhen the luminance of the image is saturated (step S132: YES), next, thecontroller 108 instructs the computational imaging camera 101 to capturean image again by shortening the exposure time (step S133). In contrast,when the maximum luminance value is equal to or less than the upperlimit value Th1 (step S132: NO), next, the computational imaginginformation determination unit 110 determines whether the light emitter109 has luminance buried in noise by checking whether the first imageacquired by the computational imaging information acquisition unit 103has a maximum luminance value less than a lower limit value Th2 (stepS134).

When the maximum luminance value is less than the lower limit value Th2,or when the luminance of the light emitter 109 is buried in noise (stepS134: YES), next, the controller 108 instructs the computational imagingcamera 101 to capture an image again by lengthening the exposure time(step S135). In contrast, when the maximum luminance value is more thanor equal to the lower limit value Th2 (step S134: NO), next, thecomputational imaging information determination unit 110 determines thatthe first image acquired by the computational imaging informationacquisition unit 103 has sufficiently high image quality with currentexposure time. In this case, the controller 108 instructs the lightemitter 109 to be turned off, and instructs the computational imagingcamera 101 to capture an image with the current exposure time. As aresult, the computational imaging information acquisition unit 103acquires a second image when the light emitter 109 is turned off (stepS136). The controller 108 may also control the exposure time of thecomputational imaging camera 101 so that the acquired second image has amaximum luminance value within a predetermined range as with the firstimage described above.

As a matter of course, the controller 108 may change setting other thanthe exposure time of the computational imaging camera 101. For example,gain may be changed.

FIG. 14 is a flowchart illustrating a procedure of main processing ofthe image identification system 12. The flowchart illustrates a flow ofprocessing before and after processing of determining image quality withthe computational imaging information determination unit 110.

When the maximum luminance value exceeds the upper limit value Th1, orwhen the luminance of the image is saturated in the determination instep S132 (step S132: YES), next, the controller 108 instructs thecomputational imaging camera 101 to capture an image again by reducingthe gain (step S137).

When the maximum luminance value is less than the lower limit value Th2,or when the luminance of the light emitter 109 is buried in noise in thedetermination in step S134 (step S134: YES), next, the controller 108instructs the computational imaging camera 101 to capture an image againby increasing the gain (step S138).

The controller 108 may control luminance of the light emitter 109instead of the exposure time or gain of the computational imaging camera101. That is, when the computational imaging information determinationunit 110 determines that the luminance of the light emitter 109 issaturated, the controller 108 controls the light emitter 109 to reducethe luminance. Conversely, when the computational imaging informationdetermination unit 110 determines that the luminance of the lightemitter 109 is buried in noise, the controller 108 controls the lightemitter 109 to increase the luminance. Increasing the luminance of thelight emitter 109 increases a luminance difference from the noise.

When the computational imaging information determination unit 110determines that the difference image has image quality less than theallowable value, the controller 108 may select another light emitterexisting in the target area and instruct the other light emitter to emitlight and turn off the light. This configuration is effective for alight source having directivity because image quality inevitablydeteriorates depending on a positional relationship between thecomputational imaging camera 101 and the light emitter 109 when thelight source having directivity is used.

The image identification system 12 according to the present embodimentcauses the controller 108 to perform re-imaging control on thecomputational imaging camera 101 when the difference image has imagequality less than the allowable value, and thus enables acquiring adifference image using a point light source in which a luminance valueis appropriately adjusted. As a result, appropriate computationalimaging information of the computational imaging camera 101 can beacquired.

The controller 108 performs the re-imaging control to correct at leastone of the exposure time and the gain of the computational imagingcamera 101, and thus enables acquiring the difference image using thepoint light source in which the luminance value of is appropriatelyadjusted.

Fourth Embodiment

FIG. 15 is a schematic diagram illustrating a configuration of an imageidentification system 13 according to a fourth embodiment of the presentdisclosure. FIG. 15 illustrates the same components as those in FIG. 1that are denoted by the same corresponding reference numerals as thosein FIG. 1 , and that are not described. The image identification system13 includes a learning device 23 including a storage unit 112 thatstores multiple learned image identification models, and a modelselector 111 that selects one image identification model from among themultiple image identification models. The learning device 23 of theimage identification system 13 includes the model selector 111 insteadof a learning unit 105 learning a learning database 102 corrected by adatabase correction unit 104, and selects an optimal imageidentification model corresponding to computational imaging informationof a computational imaging camera 101 from multiple image identificationmodels learned in advance. For example, when multiple types of amulti-pinhole mask 301 a different in a mask pattern are prepared inadvance as described above, an image identification model learned usinga captured image in a state with each of the multi-pinhole mask 301 amounted is created in advance, and then multiple image identificationmodels created in advance are stored in the storage unit 112. The modelselector 111 selects one image identification model corresponding to thecomputational imaging information of the computational imaging camera101 from among the multiple image identification models stored in thestorage unit 112.

FIG. 16 is a flowchart illustrating a procedure of main processing ofthe learning device 23 of the image identification system 13. Theflowchart illustrates a flow of processing in which the model selector111 selects an image identification model.

First, the computational imaging information acquisition unit 103acquires computational imaging information of the computational imagingcamera 101 (step S201).

Next, the model selector 111 selects one image identification modelcorresponding to the computational imaging information acquired by thecomputational imaging information acquisition unit 103 from among themultiple image identification models stored in the storage unit 112(step S211). In this processing, image identification models learnedwith various types of computational imaging information may be preparedin advance, and an image identification model learned with computationalimaging information closest to the acquired computational imaginginformation may be selected.

The image identification model selected as described above is suitablefor the computational imaging camera 101. The selected imageidentification model is set in an identification unit 106 as an imageidentification model used by the identification unit 106. Theidentification unit 106 can perform highly accurate identificationprocessing by using the image identification model.

The image identification system 13 according to the present embodimentcauses the learning device 23 to select one image identification modelcorresponding to the computational imaging information of thecomputational imaging camera 101 from among the multiple learned imageidentification models. Thus, the learning device 23 does not need tonewly perform learning, thereby enabling reduction in processing load ofthe learning device 23, and also earlier start of operation of theidentification device 30.

INDUSTRIAL APPLICABILITY

The learning method and the identification method according to thepresent disclosure are particularly useful for an image identificationsystem in an environment requiring privacy protection of a subject.

1. A learning method comprising, by an information processing deviceserving as a learning device: acquiring computational imaginginformation of a first camera that captures an image with blurring, thecomputational imaging information being a difference image between afirst image and a second image that are captured by the first camera,the first image including a point light source in a lighting state andthe second image including the point light source in a non-lightingstate; acquiring a third image captured by a second camera that capturesan image without blurring or an image with blurring smaller than that ofthe first camera, and a correct answer label assigned to the thirdimage; generating a fourth image with blurring based on thecomputational imaging information and the third image; and performingmachine learning using the fourth image and the correct answer label tocreate an image identification model for identifying an image capturedby the first camera.
 2. The learning method according to claim 1,wherein the first camera is any one of: a coded aperture cameraincluding a mask having a mask pattern with a transmittance differentfor each region; a multi-pinhole camera including a mask having a maskpattern in which multiple pinholes are formed and an image sensor with alight receiving surface on which the mask is disposed; and a light-fieldcamera that acquires a light field from a subject.
 3. The learningmethod according to claim 1, wherein the first camera includes nooptical system that forms an image of light from a subject on an imagesensor.
 4. The learning method according to claim 2, wherein the mask ischangeable to another mask having a different mask pattern.
 5. Thelearning method according to claim 1, wherein the computational imaginginformation is any one of a point spread function and a light transportmatrix.
 6. The learning method according to claim 1, wherein theinformation processing device performs lighting control of the pointlight source and imaging control of the first image captured by thefirst camera, and performs non-lighting control of the point lightsource and imaging control of the second image captured by the firstcamera.
 7. The learning method according to claim 6, wherein theinformation processing device performs re-imaging control of the firstimage and the second image captured by the first camera when thedifference image has image quality less than an allowable value.
 8. Thelearning method according to claim 7, wherein the information processingdevice corrects at least one of exposure time and gain of the firstcamera in the re-imaging control to cause each of the first image andthe second image to have a maximum luminance value within apredetermined range.
 9. An image identification method comprising: in anidentification device including an identification unit, allowing theidentification unit to receive an image captured by a first camera thatcaptures an image with blurring; identifying, by the identificationunit, the received image based on a learned image identification model;and outputting a result of the identification of the identificationunit, wherein the image identification model is the image identificationmodel created by the learning method according to claim
 1. 10. Alearning device comprising: an acquisition unit that acquirescomputational imaging information of a first camera that captures animage with blurring, the computational imaging information being adifference image between a first image and a second image that arecaptured by the first camera, the first image including a point lightsource in a lighting state and the second image including the pointlight source in a non-lighting state; a storage unit that stores a thirdimage captured by a second camera that captures an image withoutblurring or an image with blurring smaller than that of the firstcamera, and a correct answer label assigned to the third image; an imagegenerator that generates a fourth image with blurring based on thecomputational imaging information acquired by the acquisition unit andthe third image read out from the storage unit; and a learning unit thatperforms machine learning using the fourth image generated by the imagegenerator and the correct answer label read out from the storage unit tocreate an image identification model for identifying an image capturedby the first camera.
 11. An image identification system comprising: anacquisition unit that acquires computational imaging information of afirst camera that captures an image with blurring, the computationalimaging information being a difference image between a first image and asecond image that are captured by the first camera, the first imageincluding a point light source in a lighting state and the second imageincluding the point light source in a non-lighting state; a storage unitthat stores a third image captured by a second camera that captures animage without blurring or an image with blurring smaller than that ofthe first camera, and a correct answer label assigned to the thirdimage; an image generator that generates a fourth image with blurringbased on the computational imaging information acquired by theacquisition unit and the third image read out from the storage unit; alearning unit that performs machine learning using the fourth imagegenerated by the image generator and the correct answer label read outfrom the storage unit to create an image identification model; anidentification unit that identifies an image captured by the firstcamera based on the image identification model created by the learningunit; and an output unit that outputs an identification result of theidentification unit.